Not sure if this is feasible but we had an interesting scenario on https://bugzilla.mozilla.org/show_bug.cgi?id=1719850#c4

The patch touched a few tests from the mochitest-remote suite https://searchfox.org/mozilla-central/rev/5227b2bd674d49c0eba365a709d3fb341534f361/taskcluster/ci/test/mochitest.yml#807

This test-suite is currently not running on Fission (https://searchfox.org/mozilla-central/rev/5227b2bd674d49c0eba365a709d3fb341534f361/taskcluster/ci/test/mochitest.yml#816), because RemoteAgent is not Fission compatible. However the TV jobs that were scheduled on autoland after the patch from Bug 1719850 landed tried to run the mochitest-remote tests on Fission.

This led to a series of regression bugs filed, for Fission failures which are expected. Is there a way to avoid running TV tests on platforms where the original test suite is explicitly not running?

This would be nice to fix, but doesn't need to block Fission MVP.

When running a test without e10s, Fission will purposely crash with a message about requiring e10s.

Just realised that the configuration from https://searchfox.org/mozilla-central/rev/5227b2bd674d49c0eba365a709d3fb341534f361/taskcluster/ci/test/mochitest.yml#807 is probably difficult to pick up when scheduling TV jobs.

I guess we should add explicit skip-if in the browser.ini files for the suites which shouldn't run on fission. Meaning all browser.ini files under remote/cdp/test/browser: https://searchfox.org/mozilla-central/search?q=&path=remote%2Fcdp%2Ftest%2Fbrowser%2F**browser.ini&case=false&regexp=false

(In reply to Julian Descottes [:jdescottes] from comment #2)

Just realised that the configuration from https://searchfox.org/mozilla-central/rev/5227b2bd674d49c0eba365a709d3fb341534f361/taskcluster/ci/test/mochitest.yml#807 is probably difficult to pick up when scheduling TV jobs.

I guess we should add explicit skip-if in the browser.ini files for the suites which shouldn't run on fission. Meaning all browser.ini files under remote/cdp/test/browser: https://searchfox.org/mozilla-central/search?q=&path=remote%2Fcdp%2Ftest%2Fbrowser%2F**browser.ini&case=false&regexp=false

Surprisingly we still have the issue even after skipping fission in the browser.ini. TV jobs keep getting scheduled for tests explicitly skipped for Fission. For instance https://bugzilla.mozilla.org/show_bug.cgi?id=1733719

Joel, could someone help to get this fixed? Thanks!

redirecting to :mtabara who manages our team

A few notes from...

https://firefoxci.taskcluster-artifacts.net/Wi0-GKPxT6GHa_X4_hrhLw/0/public/logs/live_backing.log

executing ['c:\\mozilla-build\\python3\\python3.exe', '-u', 'mozharness\\scripts\\desktop_unittest.py', '--cfg', 'mozharness\\configs\\unittests\\win_unittest.py', '--verify', '--setpref=fission.autostart=true', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--enable-webrender', '--setpref=layers.d3d11.enable-blacklist=false', '--download-symbols', 'ondemand', '--total-chunk=2', '--this-chunk=1']
...
[task 2021-10-01T19:34:09.185Z] 19:34:09     INFO - Per-test run using mozinfo: {'os': 'win', 'processor': 'x86', 'version': '10.0.19041', 'os_version': StringVersion ('10.0'), 'bits': 32, 'has_sandbox': True, 'webrender': False, 'automation': True, 'service_pack': '', 'apple_silicon': False, 'apple_catalina': False, 'win10_2004': True, 'appname': 'firefox', 'artifact': False, 'asan': False, 'bin_suffix': '.exe', 'buildapp': 'browser', 'buildtype_guess': 'opt', 'cc_type': 'clang-cl', 'ccov': False, 'crashreporter': True, 'datareporting': True, 'debug': False, 'devedition': False, 'early_beta_or_earlier': True, 'healthreport': True, 'mozconfig': '/builds/worker/checkouts/gecko/.mozconfig', 'nightly_build': True, 'non_native_theme': True, 'normandy': True, 'official': True, 'pgo': False, 'platform_guess': 'win32', 'release_or_beta': False, 'require_signing': False, 'stylo': True, 'sync': True, 'telemetry': False, 'tests_enabled': True, 'toolkit': 'windows', 'topsrcdir': '/builds/worker/checkouts/gecko', 'tsan': False, 'ubsan': False, 'updater': True, 'e10s': True, 'headless': False, 'verify': True}
...
[task 2021-10-01T19:36:34.501Z] 19:36:34     INFO - Per-test file(s) for 'mochitest-remote': ['remote/cdp/test/browser/page/browser_domContentEventFired.js', 'remote/cdp/test/browser/page/browser_createIsolatedWorld.js', 'remote/cdp/test/browser/page/browser_frameStoppedLoading.js', 'remote/cdp/test/browser/page/browser_frameStartedLoading.js', 'remote/cdp/test/browser/page/browser_loadEventFired.js', 'remote/cdp/test/browser/page/browser_frameAttached.js', 'remote/cdp/test/browser/page/browser_frameDetached.js']
[task 2021-10-01T19:36:34.501Z] 19:36:34     INFO - Running command: ['Z:\\task_163311634435978\\build\\venv\\Scripts\\python', '-u', 'Z:\\task_163311634435978\\build\\tests\\mochitest\\runtests.py', '--enable-webrender', '--setpref=fission.autostart=true', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--setpref=layers.d3d11.enable-blacklist=false', '--appname=Z:\\task_163311634435978\\build\\application\\firefox\\firefox.exe', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Y-3hOVLuSmeYje0vzRTIfw/artifacts/public/build/target.crashreporter-symbols.zip', '--certificate-path=tests/certs', '--quiet', '--log-raw=Z:\\task_163311634435978\\build\\blobber_upload_dir\\mochitest-remote-test1_raw.log', '--log-errorsummary=Z:\\task_163311634435978\\build\\blobber_upload_dir\\mochitest-remote-test1_errorsummary.log', '--screenshot-on-fail', '--cleanup-crashes', '--marionette-startup-timeout=180', '--log-raw=-', '--flavor=browser', '--subsuite=remote', '--verify-max-time=900', '--verify', 'remote/cdp/test/browser/page/browser_domContentEventFired.js'] in Z:\task_163311634435978\build
...
[task 2021-10-01T19:36:34.871Z] 19:36:34     INFO - :::
[task 2021-10-01T19:36:34.886Z] 19:36:34     INFO - ::: Running test verification step "1. Run each test 10 times in one browser."...
[task 2021-10-01T19:36:34.887Z] 19:36:34     INFO - :::
[task 2021-10-01T19:36:34.888Z] 19:36:34     INFO - These variables are available in the mozinfo environment and can be used to skip tests conditionally:
...
[task 2021-10-01T19:36:34.894Z] 19:36:34     INFO -     e10s: True
[task 2021-10-01T19:36:34.894Z] 19:36:34     INFO -     fission: True
...
[task 2021-10-01T19:36:34.905Z] 19:36:34     INFO -     verify: True
[task 2021-10-01T19:36:34.905Z] 19:36:34     INFO -     verify_fission: False
...

I think the main problem is that the per-test mozinfo does not have 'fission' set. There is some per-test mozinfo manipulation at
https://searchfox.org/mozilla-central/rev/9bc5dcea99c59dc18eae0de7064131aa20cfbb66/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#324
Maybe that should be expanded for fission, either based on mozharness configuration, or --setprefs?

I also wonder about the runtests.py mozinfo['verify_fission'] setting, but I'm not very familiar with that and its probably not relevant to this bug.

https://hg.mozilla.org/mozilla-central/rev/4de2b96da69b

Setting status-firefox94=wontfix because we don’t need to uplift this test fix to Beta 94.